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Introduction 


•  NPS  conducted  research  in  a  proof-of-concept  case 
study  to  quantify  the  potential  benefits  of  Open 
Architecture  on  AEGIS. 

•  Knowledge-value  Added/Real-options  (KVA+RO) 
framework  was  applied  to  sustaining  engineering  in 
AEGIS  software  maintenance  and  upgrade  process. 

•  KVA+RO  framework  provides  decision-makers  with 
systematic  approach  for  analyzing  benefits  and 
assessing  risks  of  potential  IT  acquisitions. 

•  Research  conducted  from  acquirer  and  system- 
developer  perspective  to  provide  comprehensive  view 
of  entire  system  development  lifecycle;  prior  research 
conducted  from  war-fighter  perspective. 


Knowledge  Value  Added  and  Real  Options  Analysis 


•  Measures  value  and  cost  of  human  and  IT  assets. 

•  Uses  a  “market  comparables”  valuation  technique,  to 
establish  revenue  surrogates  for  discounted  cash  flow 
estimates. 

•  Allows  for  use  of  powerful  financial  metrics  in  forecasting 
value  of  strategic  options  for  replacing  systems. 

•  Estimates  value  and  risk  of  strategic  options  using  real 
options  analysis  (Hammer,  2007  measures  drivers  of 
value  and  risk). 


KVA+RO  Valuation  Framework 
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Case  Study  Methodology 


•  Analysis  first  conducted  for  one  ship  and  then  scaled  up  to  include 
entire  84  AEGIS  fleet  of  ships. 

•  Data  used  in  analysis  derived  from  interviews  with  Subject-matter 
Experts  (SME’s),  surveys  and  secondary  research. 

•  KVA  methodology  was  first  applied  in  Phase  1  of  research  under  two 
scenarios:  As-is  and  To-be. 

•  Real-options  analysis  was  conducted  in  Phase  2  on  several  three 
scenarios  to  assess  risks  associated  with  potential  strategies  for 
AEGIS  software  maintenance  and  upgrade  process. 


The  Challenge 


•  Software  is  an  increasingly  important  functionality  in  Naval  systems. 

•  For  example,  the  size  of  the  DDG  1000  combat  system  is  expected  to 
increase  35%  to  almost  1 .8  MSLOC — larger  than  the  AEGIS  Baseline  7.1  R 


Size  of  Typical  Naval  Combat  System 


3 


25 


20 


■S  IS 
~  10 
I  5 


AeglB  7.1R  DD<5  10OO 

Surface  Ship 
Combart  S'ystems 


F.'A  10  E.'F 


JSF 


Aircraft  Systems 


Source:  Horvitz,  E.,  Katz,  D.J.,  Rumpf,  R.L.,  Shrobe,  H.,  Smith,  T.B.,  Webber,  G.E.,  Williamson,  W.E.,  Winston,  P.H.,  Wolbarsht, 
James  L.,  (2006,  July).  , Software  Intensive  Systems,  ,  United  States  Naval  Research  Advisory  Committee  (NRAC)  Panel  on 
SOFTWARE  INTENSIVE  SYSTEMS,  [www.onr.navy.mil/nrac/docs/2006_rpt_software_intensive_systems.pdf]  Retrieved  July  15, 
2007. 


AEGIS  Software  Maintenance  and  Upgrade  Process 


•  AEGIS  software  maintenance  and  upgrade  process 
very  complex  involving  large  number  of  processes  in 
four  main  phases  (requirements  definition,  design,  test 
and  implementation/installation). 

•  Software  maintenance  and  upgrade  process  involves 
many  sub-processes  in  each  one  of  its  main 
processes. 

•  Entire  AEGIS  software  upgrade  lifecycle  is  intended  to 
take  18  months;  typically  takes  closer  to  24  months  due 
to  problems  found  during  testing  phase  or  certification 
failures. 


AEGIS  Software  Maintenance  and  Update  Process 

On-Ship  and  Off-Ship 
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Conducted  aboard  AEGIS-equipped  US  Naval  vessels  by 
Surface  Warfare  fleet  personnel,  various  support  personnel 
(including  contractors). 

Problems  not  found  in  testing  phase  of  the  process, 
installation  and  on-ship  testing  of  the  fielded  AEGIS  software 
update  are  identified. 

Two  different  departments  detect  AEGIS  equipment  and 
software  failures  and  analyze  effects  on  mission  capability. 
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OFF  SMIP  —  NSWC  DAHLGREN.  VA 


Conducted  at  the  Naval  Surface  Warfare  Center, 
Dahlgren,  VA. 

Primarily  deals  with  three  phases  (requirements- 
definition,  design  and  testing). 
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AEGIS  Off-Ship  Sub-Processes 


Requirements/ 
Definitions  Phase 


Design  Phase 


Test  Phase 


KVA  Methodology  Process  Steps 


1 .  Identify  core  processes  and  sub-processes. 

2.  Establish  common  units  and  level  of  complexity  to  measure  learning 
time. 

3.  Calculate  learning  time  (i.e.,  knowledge  surrogate)  to  execute  each 
sub-process. 

4.  Designate  sampling  time  period  long  enough  to  capture  representative 
sample  of  the  core  processes’  final  product  or  services  output. 

5.  Multiply  learning  time  for  each  sub-process  by  number  of  times  sub¬ 
process  executes  during  sample  period. 

6.  Calculate  cost  to  execute  knowledge  (learning  time  and  process 
instructions)  to  determine  process  costs. 

7.  Calculate  ROK  (ROK=  Revenue/Cost)  and  ROI  (ROK=  Revenue- 
Cost/Cost). 


Case  Study  Results:  Potential  Impact  of  OA 


Current  Processes 


Revised  Processes 


•  Software  Anomaly  Detected 


•  Cause  of  Anomaly  Determined 


•  Software  Bug  Report  Submitted 


•  New  Software  Version  Fielded  to  Units 


•  Remote  Diagnostics 
Detect/Fix  Anomaly 


•  Remote  Diagnostics 
Submit  Software  Bug 
Report  for  Anomaly 


Case  Study  Results:  Revised  Processes 
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Case  Study  Results:  Cost,  ROI  and  Revenues 


•  Costs  for  one  ship  decrease  $365,105;  costs  for  all 
ships  decrease  by  $26,543,825  per  year. 

•  Return  on  investment  for  one  ship  increases  from  69% 
to  789%;  ROI  for  all  ships  increases  from  320%  to 
72,287%. 

•  Revenues  (benefits)  for  one  ship  increase  $2,488,179 
to  $3,837,931 ;  revenues  for  all  ships  increase 
$209,007,032  to  $322,386,181. 


Case  Study  Results:  Costs  Savings 


Costs  for  ONE  Ship 


Costs  for  ALL  Ships 


Process! Revised  Process 

As-is 

To-be 

Potential 

Savings 

Software  Anomaly  Detected 

New  Release  Fielded  (Push  to 
Ship  via  Distance  Support) 

$14,301 

$7,150 

Cause  of  Anomaly  Determined 

Remote  Diagnostics 

Detect/Fix  Anomaly 

$251,597 

$50,319 

Software  Bug  Report  Submitted 

Remote  Diagnostics  Submit 
Software  Bug  Report  for 
Anomaly 

$1,430 

$1,430 

Anomaly  Verified 

$17,021 

$17,021 

Anomaly  Appended  to  Working 

List  of  Known  Issues 

$1,307 

$1,307 

Workaround  Developed 

$17,021 

$17,021 

New  Software  Version 

Developed 

$236,750 

$236,750 

Known  Anomalies  are  Resolved 

$100,639 

$100,639 

New  Software  Version  Fielded 
to  Units 

$156,840 

$163 

Totals 

$796,907 

$431,802 

$365,105 

Process/Revised  Process 

As-is 

To-be 

Potential 

Savings 

Software  Anomaly  Detected 
New  Release  Fielded  (Push 
to  Ship  via  Distance 
Support) 

$14,301 

$7,150 

Cause  of  Anomaly 

Determined 

Remote  Diagnostics 
Detect/Fix  Anomaly 

$251,597 

$50,319 

Software  Bug  Report 
Submitted 

Remote  Diagnostics  Submit 
Software  Bug  Report  for 
Anomaly 

$1,430 

$1,430 

Anomaly  Verified 

$17,021 

$17,021 

Anomaly  Appended  to 
Working  List  of  Known 
Issues 

$1,307 

$1,307 

Workaround  Developed 

$17,021 

$17,021 

New  Software  Version 
Developed 

$236,750 

$236,750 

Known  Anomalies  are 
Resolved 

$100,639 

$100,639 

New  Software  Version 

Fielded  to  Units 

26,349,120 

$13,723 

Totals 

$26,989,187 

$445,362 

$26,543,825 

Case  Study  Results:  ROI 


ROI  For  ONE  Ship 


ROI  For  ALL  Ships 


Process/Revised  Process 

As-is 

To-be 

Software  Anomaly  Detected 

New  Release  Fielded  (Push  to 
Ship  via  Distance  Support) 

297% 

3874% 

Cause  of  Anomaly  Determined 
Remote  Diagnostics  Detect/Fix 
Anomaly 

236% 

4605% 

Software  Bug  Report  Submitted 
Remote  Diagnostics  Submit 
Software  Bug  Report  for 
Anomaly 

2108% 

39636% 

Anomaly  Verified 

234% 

234% 

Anomaly  Appended  to  Working 
List  of  Known  Issues 

1188% 

1188% 

Workaround  Developed 

78% 

78% 

New  Software  Version 

Developed 

-36% 

-36% 

Known  Anomalies  are  Resolved 

-41% 

-41% 

New  Software  Version  Fielded  to 
Units 

-36% 

185408% 

Totals 

69% 

789% 

Process/Revised  Process 

As-is 

To-be 

Software  Anomaly  Detected 

New  Release  Fielded  (Push  to 
Ship  via  Distance  Support) 

33278% 

333681% 

Cause  of  Anomaly  Determined 
Remote  Diagnostics  Detect/Fix 
Anomaly 

28133% 

395159% 

Software  Bug  Report  Submitted 
Remote  Diagnostics  Submit 
Software  Bug  Report  for 

Anomaly 

185334% 

3337713% 

Anomaly  Verified 

27943% 

27943% 

Anomaly  Appended  to  Working 

List  of  Known  Issues 

108113% 

108113% 

Workaround  Developed 

14856% 

14856% 

New  Software  Version 

Developed 

5277% 

5277% 

Known  Anomalies  are  Resolved 

4841% 

4841% 

New  Software  Version  Fielded  to 
Units 

-68% 

185408% 

Totals 

320% 

72287% 

Case  Study  KVA  Results:  Revenues 


Revenues  For  One  Ship 


ProcBssI Revised  Process 

As-is 

To-be 

Difference 
(As-is,  To- 
be) 

Software  Anomaly  Detected 

New  Release  Fielded  (Push  to 
Ship  via  Distance  Support) 

$56,826 

$284,131 

Cause  of  Anomaly  Determined 
Remote  Diagnostics  Detect/Fix 
Anomaly 

$845,629 

$2,367,761 

Software  Bug  Report 

Submitted 

Remote  Diagnostics  Submit 
Software  Bug  Report  for 
Anomaly 

$31,570 

$568,262 

Anomaly  Verified 

$56,826 

$56,826 

Anomaly  Appended  to 

Working  List  of  Known  Issues 

$16,837 

$16,837 

Workaround  Developed 

$30,307 

$30,307 

New  Software  Version 

Developed 

$151,537 

$151,537 

Known  Anomalies  are 

Resolved 

$59,194 

$59,194 

New  Software  Version  Fielded 
to  Units 

$101,024 

$303,073 

Totals 

$1,349,752 

$3,837,931 

$2,488,179 

Revenues  For  ALL  Ships 


ProcessI  Revised 
Process 

As-is 

To-be 

Difference 
(As-is,  To-be) 

Software  Anomaly 

Detected 

New  Release  Fielded 
(Push  to  Ship  via  Distance 
Support) 

$4,773,407 

$23,867,035 

Cause  of  Anomaly 
Determined 

Remote  Diagnostics 
Detect/Fix  Anomaly 

$71,032,841 

$198,891,956 

Software  Bug  Report 
Submitted 

Remote  Diagnostics 

Submit  Software  Bug 

Report  for  Anomaly 

$2,651,893 

$47,734,069 

Anomaly  Verified 

$4,773,407 

$4,773,407 

Anomaly  Appended  to 
Working  List  of  Known 
Issues 

$1,414,343 

$1,414,343 

Workaround  Developed 

$2,545,817 

$2,545,817 

New  Software  Version 
Developed 

$12,729,085 

$12,729,085 

Known  Anomalies  are 
Resolved 

$4,972,299 

$4,972,299 

New  Software  Version 
Fielded  to  Units 

$8,486,057 

$25,458,170 

$209,007,032 

Real-options  Analysis 


1.  List  of  projects  and 
strategies  to  evaluate 
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2.  Base  case  projections  for  each 
project 


Historical  Data  Gathering 


Start  with  a  list  of  projects  or 
strategies  to  be  evaluated. . .  these 
projects  have  already  been 
through  qualitative  screening 


5.  Framing  Real  Options 


. .  .will  the  assistance  of  time- 
series  forecasting  and 
historical  data. . . 


6.  Options  analytics,  simulation 
and  optimization 


Simulation 


Lattice 
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chosen  for  real  options 
analysis  and  the  projector 
portfolio  real  options  are 
framed. . . 


. .  .real  options  analytics  are  calculated 
through  binomial  lattices  and  closed-form 
partial-differential  models  with 
simulation. . . 


3.  Develop  static  financial 
models  with  KVA  data 


4.  Dynamic  Monte  Carlo 
simulation 
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. .  .the  user  generates  a  traditional 
series  of  static  base  case  financial 
(discounted  cash  flow)  models  for 
each  project... 

7.  Portfolio  optimization  and  asset 
allocation 


Optimization 


C/5 

2 

. .  .sensitivity  and  scenario  analysis 
coupled  with  Monte  Carlo  simulation  is 
added  to  the  analysis  and  the  financial 
model  outputs  become  inputs  into  the  real 
options  analysis. . . 


8.  Reports  presentation  and 
update  analysis 


. .  .stochastic  optimization  is  the  next 
optional  step  if  multiple  projects  exist  that 
requires  efficient  asset  allocation  given 
some  budgetary  constraints. . .  useful  for 
strategic  portfolio  management. . . 


jg. ..create  reports,  make  decisions, 
mid  do  it  all  again  iteratively  over 
time . . . 


Real-Options  Valuation  Results:  Strategies  A-D 


Strategy  A 

Strategy  B 

Strategy  C 

Strategy  D 

STRATEGIC 

OPTION 

AS-IS 

(1) 

TO-BE 
(W/out 
Changes  in 
Fielding  to 
Units) 

TO-BE 
(1, 3,4,5, 6, 7) 

TO-BE 

(1,2,3,4,5,6,7) 

Total  Strategic 
Value 

$196M 

$995M 

$1.24B 

1.48B 

Volatility 

10% 

30% 

60% 

50% 

Total  Cost 

$208M 

$96M 

$30M 

$80M 

Key: 

1 .  As-is 


2.  Implement  remote  diagnostics/prognostics  through  ORTSTARS/Distance  support,  plus  the  ability  of  the  crew  to  inject  a 
trouble  report  through  ORTSTARS/Distance  support. 

3.  Number  2  plus  providing  software  updates  to  the  ship  on  media,  then  having  the  crew  install  with  technician  DS  (remote) 
help.  (Could  also  postulate  a  "sense  and  respond"  sort  of  thing,  in  which  a  "local"  tech  is  scheduled  to  the  ship  based  on  its 
availability  and  the  update's  arrival.  Would  count  on  local  assets,  not  travel  from  Dahlgren...) 

4.  Number  3  plus  notification  of  the  ship  that  the  update  is  available  for  download.  Ship  initiates  download  and  installs  with 
DS  help. 

5.  Number  4,  except  that  the  ship  is  notified  that  updates  are  available.  The  on-ship  operators  tell  DS  they're  ready,  and  the 
remote  tech  takes  control  and  installs  the  update. 

6.  Number  4  except  that  the  update  is  pushed  to  the  ship,  then  cached  until  operators  are  ready  to  install.  Ship  installs  with 
DS  assistance  if  needed. 

7.  Final  state  in  which  the  update  is  pushed  to  the  ship  and  installs  during  slack  time,  notifying  the  ship  and  allowing 
operators  to  say  "not  now." 


Research  Implications 


•  Results  from  our  research  indicate  that  implementing  OA  could  result  in 
substantial  cost  savings,  optimal  return  on  investment  and  increased 
benefits. 

•  Software  updates  could  be  available  via  push  or  pull  method  with  OA.  In 
the  pull  method,  user  downloads  and  installs  updates.  With  the  push 
method,  software  is  pushed  to  the  network  node  remotely,  thereby 
reducing  onsite  personnel  while  speeding  up  the  upgrade  process. 

•  New  software  updates  could  be  fielded  to  the  ship  through  in  either 
method,  resulting  in  reduced  cycle-time  fielding  new  software  to  its 
shipboard  configuration. 

•  Remote  diagnostics  could  also  be  performed  to  further  reduce  cycle-time. 


